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REMARKS 

Claims 1-16, 18-25 and 28 are pending in the case. Claims 1,13 and 25 have been 
amended. No new matter has been added to this application. 

Rejection of claims! -16, 18-25 and 28 under 35 TIS.C S103(a) 

Claims J -1 6, 1 8- 25 and 25 have been rejected under 35 U.S.C. 103(a) as being 
unpatentable over Chen, U.S. Patent No. 5,75,561 in view of Fukunaga, U.S. Patent No. 
6,346,940. The Examiner contends that Chen teaches Applicants* invention as claimed 
but correctly notes that Chen does not teach or disclose an instrument included in the real 
view of the environment, determining a preferred path for positioning of said instrument 
and inserting the instrument in the graphics guide. The Examiner contends that Fukunaga 
teaches including in the real image the instrument for the medical procedure. The 
Examiner argues that it would have been obvious to include the real image of the 
instrument in the environment in the Chen system. Applicants respectfully traverse the 
rejection. 

The present invention is directed to a method and system for augmented reality 
guided instrument positioning. A real view of an environment is displayed. A preferred 

00 

path for positioning an instrument is determined. The preferred path is marked with a m 

(I) 

graphics guide. The real view is augmented with a rendering of the graphics guide such _ j 
that at least one portion of the graphics guide is transparent with respect to other portions ^ 
of the graphics guide to provide a substantially unobstructed view through the at least one ^ 
portion of the graphics guide to at least a portion of the instrument to facilitate alignment ^ 
of the real instrument with the virtual guide. The instrument is aligned to the graphics 00 
guide so that the instrument appears in a same location as the graphics guide in the fTI 
augmented view thereby creating a mutual spatial relationship between the real O 
instrument and the virtual guide. When properly aligned, the instrument is visible -£) 



through the at least one transparent portion of the graphics guide. The instrument is then 
inserted along the length of the graphics guide. The real instrument and virtual graphics 
guide appear in the same view for precise spatial alignment. 



-< 
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Chen discloses a surgical targeting system for facilitating the location of a 
particular anatomical structure during a medical procedure, A real image and a virtual 
image are obtained. The virtual image can be superimposed over the real image and the 
virtual image can be modified to expose virtual markers that highlight various anatomical 
structures to the physician. However, unlike the present invention, Chen does not teach 
or disclose marking a preferred path with a graphic guide. Nor does Chen teach or 
disclose modifying a virtual guide to expose the real image of the instrument thereby 
facilitating alignment of the real instrument. Chen's manipulation of the virtual graphics 
is used to modify the view of the existing virtual graphics in itself and not to expose a 
mutual spatial relationship between the real instrument and the virtual guide to assist in 
alignment of the real instrument as recited in amended claims 1, 13 and 25. 

Furthermore* Chen discloses tracking a virtual image of an instrument as it moves 

through an anatomical structure when some or the entire instrument might be hidden 

from the view of the camera. In fact Chen indicates in col. 11, line 60 - col. 12, line 4, 

that a line can be extended along the axis of the surgical instrument to show the physician 

where the instrument would go if moved further along the current trajectory* Then Chen 

states that "Of course, since the virtual image generated by the video-based surgical 

targeting system will then differ significantly from the real image generated by the video 

camera 45, video mixing device 35 should be directed to totally suppress the real image ft] 

CO 

generated by video camera 45 so that it will no longer be shown on display 40 f in order to 
avoid confusing the physician/* Applicants submit that not only does Chen not teach or ^ 
disclose Applicants' invention, but in fact teaches away from Applicants' invention. ^ 
Chen moves to using purely virtual images for guiding a surgical procedure, requiring ^ 
instrument tracking, and explicitly states that the real image should be suppressed to J2 
avoid confusion. On the contrary, the present invention combines images of virtual guide Tl 
and real instrument to assist the physician in guiding the instrument, without the need for ^ 
an external tracking system. The present invention uses the virtual guide to directly 
support precise spatial alignment of the instrument. Applicants respectfully submit that 
Chen does not teach or disclose Applicants' invention as claimed. 

Fukunaga teaches that a guiding marker preparation unit forms guiding markers 
(see col. 7, lines 65 to col. 8 line 3). Fukunaga does not teach or suggest "augmenting the 
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real view with a rendering of the graphics guide" as claimed in claims 1 and 13 and 25, 
Fukunaga teaches an image processing system installed adjoining an electronic 
endoscope system (see col. 5 t lines 30-35). Image processing system views are shown on 
a display unit while views from the electronic endoscope are shown on a separate monitor 
(see Figure 3). Fukunaga does not teach or suggest an augmenting the real view with a 
rendering of the graphics guide, essentially as claimed in claims 1, 13 and 25. Fukunaga 
strictly keeps the virtual and the real images on separate monitors. Fukunaga onjy 
prepares a "map" by inserting guiding markers into a virtual model, similar to putting 
signpost up at road forks where the only decision to be made is whether to go right or 
left. Fukunaga does not teach showing a real view that includes the instrument to be 
positioned (in Fukunaga, the instrument is an endoscope and captures the real view, thus 
not being visible itself), and Fukunaga does not teach to overlay the real view with the 
virLual map for precise spatial alignment (which, of course, cannot be done as the 
endoscopic instrument is not visible in the real view). 

Claims 1,13 and 25 are believed to be allowable for additional reasons. Fukunaga 
teaches that a degree of transparency of the virtual endoscopic image Ai is preset, but 
may be adjusted by an operator (see col. 1 1, lines 44-49). Fukunaga does not teach or 
suggest "at least one portion of the graphics guide is transparent with respect to other 
portions of the graphics guide to provide a substantially unobstructed view through the at 
least one portion of the graphics guide to at least a portion of the instrument" as claimed 
in claims 1, 13 and 25. As the Examiner has suggested with respect to claim 3, Fukunaga ni 
fails to teach or suggest a transparent guide marker. Fukunaga treats the guiding markers 
separately from the endoscopic image Ai. Fukunaga teaches that guiding markers are 
placed on the virtual endoscopic image Ai (see for example, col. 18, line 38 to col. 19, 2> 
line 60). The transparency of the virtual endoscopic image Ai is set before the guiding [> 
markers are placed (see col. 19, lines 30-35). Thus, Fukunaga does not teach or suggest 
transparent guiding markers. Indeed, Fukunaga fails to teach or suggest that anything 
other than certain virtual endoscopic images (e.g., images Hi as shown in Figure 15a) q 
may be semitransparent. 

In Fukunaga, the guide markers support the user to find the right way along a 
branching system of "channels" (e.g., bronchial tree). The guide markers are not 
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intended to mark exact spatial endoscope locations inside the channels* In contrast, the 
present invention uses the virtual graphics guide in the context of an augmented view to 
precisely mark where a real instrument has to be placed. The user has to perceive the 
exact spatial relationship between the virtual graphics guide and the real instrument, both 
visible in the augmented view, and bring the instrument (e.g., needle) into correct 
alignment with the graphics guide. 

Furthermore, Fukunaga fails to teach or disclose "aligning the instrument to the 
graphics guide so that the instrument appears in a same location as the graphics guide in 
the augmented view, and when properly aligned, the instrument is visible through the at 
least one transparent portion of the graphics guide; and inserting the instrument along the 
length of the graphics guide"- Fukunaga describes a variety of visualization options. To 
see behind the surface of the virtual model or beyond a graphics guide, strictly within the 
realm of a pure virtual image, Fukunaga renders the guide semi-transparent or as a 
wireframe. - 

Even if transferred into an augmented view, the simple visualization options of 
Chen and Fukunaga are not sufficient to correctly perceive the alignment of a real 
instrument and a graphics guide in the augmented view. The core problem is that of 
incorrect occlusion. The virtual guide always occludes the real instrument; correctly 
when the real instrument is indeed behind the guide, but also incorrectly when the real 
instrument is actually in front of the virtual guide. Semitransparent rendering of the 
guide avoids complete occlusion of the real instrument; however, even the partial 
occlusion of the real instrument by the virtual guide is wrong when the real instrument is 
in front of the guide, and still interferes with the correct spatial perception. 

The present invention introduces the concept of particular virtual guides. Along 
the length of the guide, the transparency of the guide is modulated. By this it is meant 
that there are sections where the guide "disappears" to provide an essentially 
unobstructed view onto the instrument during the alignment procedure, and other sections 
where the guide is clearly visible. This modulation introduces a completely new quality 
to the visualization of the virtual guide, and to the perception of the alignment of the real 
instrument and virtual guide in the augmented view. 
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Both Chen and Fukunaga fail to teach or suggest an augmented view as an 
overlay of a real and a virtual view, a transparent guide marker, modulation of a guide 
marker's transparency, and alignment of a real instrument and a virtual graphics guide. In 
light of Chen's and Fukunaga' s deficiencies, one of ordinary skill in the art would not be 
lead to augmenting the real view with a rendering of the graphics guide having at least 
one portion of the graphics guide being transparent with which the real instrument is 
aligned, essentially as claimed in claims 1,13 and 25. Therefore, both Chen and 
Fukunaga, whether taken alone or in combination, fail to teach or suggest all the 
limitations of claims 1, 13 and 25, 

Claims 2-12 and 28 depend from claim 1. Claims 14-16 and 18- 24 depend from 
claim 13. The dependent claims are believed to be allowable for at least the reasons 
given for claims 1 and 13. Applicants respectfully request that the rejection of claims 1- 
16 and 18-24 under 35 U.S.C. 103(a) be withdrawn, 

- For the forgoing reasons, the present application, including claims 1-16, 18-25 
and 28, is believed to be in condition for allowance. The Examiner's early and favorable 
action is respectfully urged. 



Respectfully Submitted, 



Date: November 18,20QS 
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